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(54) Image record big/ reproducing apparatus In monitor system 

(57) For image recording/reproduction in a monitor- 
ing system, a monitoring image signal (110) as inputted 
is converted into a plurality of time-serial digitized mon- 
itoring image frames to be stored temporarily in a first 
storage area (1 73). From the plurality of digitized moni- 
toring image frames stored in the first storage area, a 
plurality of monitoring Image frames relevant to at least 
one type of concerned event for monitoring are 
extracted to be stored in a second storage area (161). A 
monitored event database having a data structure com- 
prised of elements each containing information con- 
cerning a storing location of relevant monitoring image 
frame in the second storage area and information con- 
cerning attributes of the relevant monitoring image 
frame for each of the monitoring image frames stored in 
the second storage area is generated to be stored in a 
third storage area (162). In response to a perusal 
request issued by user by using a WEB browser (501) 
and sent via a network, the monitored event database is 
searched for acquiring a storing location of plural moni- 
toring image frames relevant to the perusal request in 
the second storage area to thereby select from the sec- 
ond storage area a plurality of monitoring image frames 
relevant to the perusal request The selected plural 
monitoring image frames are converted into HTML data 
to be sent to a display device connected via a network 
to be displayed on a display device in a two-dimensional 
layout (Fig. 8) or in the form of time-serial animation 
(552). 
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Description 

BACKGROUND OF THE INVENTION 

[0001 ] The present invention relates generally to an s 
image recording/playback apparatus which is designed 
tor use in a system for monitoring various events taking 
place in a place concerned by making use of movie 
techniques. More particularly, the present invention is 
concerned with an improvement of the image record- to 
ing/playback apparatus which makes it possible to sur- 
vey and/or retrieve a monitoring picture or image 
recorded over an extended period within a short time on 
the order of one several -tenth of the recording period. 
Further, the present invention is concerned with a is 
image recording/playback method and a recording 
medium which stores the method in the form of a pro- 
gram executable with a computer. 
[0002] In a place such as supermarket conven- 
ience store, elevator hall or the like which is frequented so 
by a large number of persons and which is difficult to be 
surveyed by a person in charge of management or 
watching, a monitoring camera is installed for the pur- 
pose of preventing accidents and/or crimes in advance. 
In this conjunction, the monitoring system far monitoring zs 
the places such as mentioned above by viewing the 
moving images or pictures taken by such monitoring 
camera can generally be classified into two types, i.e.. a 
system which necessitates a constant survey of col- 
lected images picked up by the monitoring cameras by ao 
a person at a canter station (hereinafter referred to as 
the online-type monitoring system) and a system in 
which pictures or images taken by the monitoring cam- 
era are stored accumulatively for allowing survey and 
inspection thereof later as occasion requires, e.g. upon as 
commitment of offense or crime (hereinafter this type of 
monitoring system will be referred to as a storage-type 
monitoring system). The storage-type monitoring sys- 
tem involves scarcely personnel expenses when com- 
pared with the online-type monitoring system. For this to 
reason, the former is adopted widely in practical appli- 
cations. At this juncture, it is noted that in the storage- 
type monitoring system, a time-lapse video recorder is 
primarily employed for storing and playback the pictures 
or images as picked up by the monitoring camera. «s 
[0003] With the time-lapse video recorder, such a 
video recorder is meant which is modified on the basis 
of a conventional video recorder such that the time 
interval for recording the images at a rate of thirty 
frames per second is increased to an interval on the so 
order of several frames per second for recording only 
sparsely. Thus, with the time-lapse video recorder 
based on the conventional video recorder, it is possible 
to perform the recording over a period of about 180 
hours (about eight days) by recording at the rate of one ss 
frame per second by using a tape medium capable of 
recording over 120 minutes with the conventional video 
recorder. Of course, by increasing the frame interval, 



recording can be performed over a further extended 
period which will however be accompanied with dropout 
of image of some object under monitoring, to a disad- 
vantage. For this reason, the recording interval which 
exceeds one second or so is not adopted in practical 
applications. Further, it is conceivable to use a tape 
medium which can ensure a recording duration which 
exceeds 120 minutes or so. However, in consideration 
of durability (or possibility of breakage) of the tape 
medium, the tape which aDows recording over 120 min- 
utes or so is usually employed. Such being the circum- 
stances, the longest recording time duration of the time- 
lapse video recorder based on the conventional video 
recorder is approximately 180 hours. 
[0004] image recording is performed starting from 
the leading end of the tape toward the trailing end 
thereof. Upon reaching the trailing end of the tape in the 
image recording operation, tape rewinding operation is 
automatically put into effect, whereon overwriting oper- 
ation is performed, starting again from the leading end 
of the tape. In this manner, a series of images can be 
stored ordinarily over a time span of 180 hours or so at 
maximum. For perusing or surveying the images as 
recorded, substantially same method as that for the 
conventional video recorder can be resorted to. How- 
ever, when the images recorded by the time-lapse video 
recorder based on the video recorder are to be repro- 
duced straightforwardly in a same manner as the picture 
recorded by the conventional video recorder, the play- 
back speed may amount to be thirtyfold. 
[0005] Accordingly, the time-lapse video recorder is 
ordinarily provided with a mechanism for allowing the 
recorded images to be reproduced at a rate of several 
frames per second. 

[0006] Among the time-lapse video recorders, there 
has been developed such a recorder in which a digital 
recording on a hard disk is adopted. With the time-lapse 
video recorder which is based on the VHS video 
recorder, the recording head may be damaged because 
recording/stop operations are mechanically repeated. 
Besides, the tape medium is likely to become degraded, 
incurring deterioration of quality of the reproduced 
images. These problems can certainly be solved by the 
digital recorder mentioned above. Parenthetically, the 
digital recorder is generally termed the digital time- 
lapse video recorder distinctively from the time-lapse 
video recorder based on the conventional VHS video 
recorder. 

[0007] As will now be appreciated from the forego- 
ing, monitoring images can be stored over an extended 
time period with the time-lapse video recorder. How- 
ever, the time-lapse video recorder suffers serious prob- 
lems with regard to the image reproduction for the 
survey or perusal. Firstly, it has to be pointed out that 
several times as long as the recording time of the 
recording medium (120 minutes in the case of the video 
tape) will be taken for observing thoroughly all the 
images or pictures recorded during the extended period 
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of for 180 hours. Secondly, effectiveness of search 
and/or perusing (or surveying) operations is extremely 
poor because of disordered states of the objects con- 
cerned with images of significance and insignificance 
being recorded concurrently and alternately. In the 
present state of the art. reproduction is performed sim- 
ply from the leading end of trie tape while observing the 
images only rumbtingly, which wil often result in failure 
to notice the image of interest or concern, to a great dis- 



[0008] As is apparent from the above, the perusal of 
the images record ad accumulatively wtth the time-lapse 
video recorder imposes a great burden on the user 
because of necessity for concent relive observation of 
the images over a much extended time period. ra 
[0009] At this juncture, it should be added that as an 
approach for solving the problems mentioned above, 
there has already been proposed such a monitoring 
system in which an infrared sensor such as a man-sen- 
sor is combined with the time-lapse video recorder. 20 
However, such system suffers many drawbacks. Firstly, 
with the time-lapse video recorder based on the VHS 
video recorder, mechanical portion for effectuating the 
recording/stop operations will be damaged at a rela- 
tively high failure rate because of frequent on/off opera- 25 
tions. Secondly, reliability can not be ensured in respect 
to the effective coverage range of the sensor and the 
sensitivity thereof. Thirdly, high cost win be involved in 
retrofitting the monitoring system. For these reasons, 
combination of the sensor with the time-lapse video 30 
recorder is seldom adopted in the practical applications, 
leaving the aforementioned problems unsolved. 

SUMMARY OF THE INVENTION 

35 

[0010] The fundamental problem of the conven- 
tional time -lapse video recorder can be found in that the 
image data as inputted to the recorder are simply 
recorded on a time serial basis without performing any 
classification of the data. In this conjunction, let's sup- *o 
pose a case where a person monitors a certain place as 
to occurrence of some event, tn that case, he or she will 
take notes of the events such as, for example, coming- 
in/but of persons, positional changes of equipment or 
the like for classification and orderly arrangement with a as 
view to facilitating the search or retrieval of the events 
as recorded. Further, upon perusal or survey, only the 
images of concern (e.g. images showing features of 
incomers or intruders, destruction of equipment or facil- 
ities, theft of something valuable) wiD first be checked so 
briefly and thereafter examined in detail without haste. 
Contrivance of this sort w3l become more important as 
the time taken for the monitoring increases. 
[0011] It goes without saying that even in the moni- 
toring system in which the time-lapse video recorder is ss 
employed as the image recording/playback apparatus, 
capability of orderly arrangement classification and 
selection of the images which would be performed if the 



images are handled by a person, as mentioned above, 
can contribute considerably to reduction of the time 
taken for the search while alleviating significantly the 
burden imposed mentally on the user. 
[0012] As will be understood from the foregoing, the 
conventional image recording/playback apparatus typi- 
fied by the time-lapse video recorder suffers drawbacks 
in respect to the storage of images with classification 
and orderly arrangement and the surveying facility 
which allows the user to search the image of concern 
without any appreciable difficulty. 
[0013] In the Bght of the state of the art described 
above, it is an object of the present invention to provide 
an image recording/playback apparatus for a monitoring 
system which apparatus can perform recording and 
reproduction of monitoring images very effectively and 
conveniently for the user. 

[0014] More particularly, it is an object of the 
present invention to provide an image recording appara- 
tus, a monitoring system and a monitoring method 
which can detect various events occurring in a place 
subjected to the monitoring and store the images with 
classification and orderly arrangement in dependence 
on the types or species of the events while presenting 
an interface capable of reducing signBicantjy the time 
taken for the user to survey or peruse the images. 
[001 5] It is also an object of the present invention to 
provide a recording medium which stores the monitor- 
ing method in the form of a program or programs exe- 
cutable by a computer. 

[0016] In view of the above and other objects which 
will become apparent as the description proceeds, there 
is provided according to an aspect of the present inven- 
tion an image recording/playback apparatus for a moni- 
toring system which apparatus includes a unit for 
inputting and digitizing images on a time-serial basis, an 
input image displaying unit for displaying the image 
being inputted currently, a monitored event detecting 
unit for detecting a monitored event from the image as 
inputted, an image recording decision unit for deciding 
whether the image as inputted is to be stored or not on 
the basis of the event as detected, an image storing unit 
for storing an image which is decided to be recorded by 
the image recording decision unit, a monitored event 
storing unit for recording a storage location of the image 
as stored and the monitored event Information as 
detected, an image selecting/disposing unit for search- 
ing the event information which meets user's perusal 
demand and selecting the images for perusal to thereby 
lay out the event(s) together with the information 
thereof, and a monitoring information display unit for 
displaying one or a pturaBty of the images and/or the 
event information as laid out 

[0017] As the recording medium for the image stor- 
ing unit and the monitored event storing unit mentioned 
above, a hard disk, a DVD-RAM or a random access 
memory of flash-memory type may be employed. 
[0018] Further, the monitored event detecting unit 
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mentioned above may be implemented by resorting to a 
scheme which is disclosed, tor example, in the U.S. Pat- 
ent No. 5.721.692 titled "MOVING OBJECT DETECT- 
ING APPARATUS". According to this method, 
presence/absence of change of the background struc- 
ture, presence/absence of change of illumination, pres- 
ence/absence of a moving abject and the 13® are 
detected, respectively, as monitored events on the basis 
of difference degree between the images inputted upon 
sampling and an Image recorded in the past Further, a 
region of flesh color is detected. Additionally, decision is 
made as to whether or not a face candidate area or field 
exists in the scene in which a moving object is found. On 
the basis of the results of the detection and the decision 
mentioned above, monitored event such as "presence 
of a person whose face is presented most closely to the 
monitoring camera" is detected. 
[0019] In a preferred mode for carrying out the 
invention, the aforementioned image recording decision 
unit may be so designed as to decide whether or not the 
recording Is to be performed In consideration of the lat- 
est monitored event species/histories of the events 
already detected, difference between the time point at 
which the immediately preceding image was recorded 
and the current time point, residual capacities of the 
media employed In the various storage units, minimum 
time duration for storage as designated by the user. 
[0020] Further, the monitored event storing means 
mentioned previously may preferably be so designed as 
to store characteristic quantities concerning the moni- 
tored avent(s) such as the location where the image is 
to be stored in the image storing unit, date/time of 
occurrence and termination of the event, position of the 
event within the image frame, a camera identifier (or 
identifier of the place being monitored) in the case 
where decision is made that image recording is to be 
performed for the event(s). 

[0021] In another preferred mode for carrying out 
the present invention, the image selecting/disposing 
unit may be so designed as to search or retrieve the 
monitored event information in accordance with the 
user's survey request such as, for example, "a list of 
detected face images", "a list of events occurred within 
given one hour", "display in animation the images 
recorded for a given monitored event" , "a list of changes 
of installations at a place being monitored", "distribution 
of monitored events in one day" and the Eke. to thereby 
generate a screen image for a display unit by selectively 
disposing the appropriate images in a manner easy for 
viewing by the user. Parenthetically, rules/algorithms 
concerning the objects for search or retrieval and the 
image/layouts for each of the screen images may previ- 
ously be incorporated as firmware in the apparatus. 
[0022] In yet another preferred mode for carrying 
out the present invention, there may be provided addi- 
tionally a unit tor establishing synchronism among a plu- 
rality of images being displayed or monitored event 
information in accordance with surveying operation per- 



formed by the user. 

[0023] By virtue of the arrangement mentioned 
above, an interlace which enables display of plural 
images in association with one another can be realized, 
5 whereby the monitored event information as well as the 
images can be presented to the user very efficiently and 
multilaterally. 

[0024] The above and other objects, features and 
attendant advantages of the present invention will more 

io easily be understood by reading the fallowing descrip- 
tion of the preferred embodiments thereof taken, only by 
way of example, in conjunction with the accompanying 
drawings, being appreciated that many modifications 
and variations of the present invention are possible in 

is the light of the techniques disclosed herein. 

DESCRIPTION OF THE PREFERRED EMBODI- 
MENTS 

20 [0025] In the course of the description which fol- 
lows, reference is made to the drawings, in which: 

Rg. 1 is a schematic block diagram showing, by 
way of example, a system configuration of a moni- 
25 toring system to which an image recording/play- 
back apparatus according to the present invention 
can find application; 

Rg. 2 is a schematic block diagram showing, by 
way of example, a system configuration of an image 

so recording/playback apparatus according to an 
embodiment of the present invention; 
Rg. 3 is a view for illustrating a method of sampling 
basic frame images in a system for recording 
images relevant to monitored events as detected 

35 according to an embodiment of the present inven- 
tion; 

Rg. 4 is a view for illustrating a method of sampling 
frame images on the basis of presence/absence of 
a monitored event in the system for recording 
40 images relevant to monitored events as detected 
according to the present invention; 
Figs. 5A and 5B are views for illustrating a method 
of controlling dynamically a frame image sampling 
period or interval in dependence on changes of 
45 monitored event's) in the system for recording 
images relevant to monitored events as detected 
according to the present invention ; 
Rg. 6 is a view for illustrating a screen image gen- 
erated in an image recording mode of the monitor- 
so ing image recording/playback apparatus according 
to an embodiment of the present invention; 
Rg. 7 is a view for illustrating a screen image gen- 
erated in a reproduction mode of the monitoring 
image recording/playback apparatus according to 
ss an embodiment of the present invention; 

Rg. 8 is a view for aiustrating a list displaying 
screen image generated by the monitoring image 
recording/playback apparatus according to an 
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embodiment of the present invention; 
Fig. 9 is a view for illustrating a user Interface for 
enabfing manipulation of the image recording/play- 
back appar a tus for recording, reproduction and 
perusal of monitoring images at a remote place s 
such as a monitoring center; 
Fig. 1 0 is a schematic block diagram for illustrating 
a functional configuration of the image record- 
ing/playback apparatus according to an embodi- 
ment of the present invention ; 10 
Fig. 11 is a view for illustrating a method of setting 
areas or fields concerned and results of event 
detection in the areas as carried out by a monitored 
event detecting engine in the course of executing a 
procedure for detecting the monitored event is 
according to an embodiment of the present inven- 
tion; 

Fig. 1 2 is a view for illustrating a time-space image 
generated by setting the concerned areas shown in 
Fig. 1 1 in the course of executing the procedure for 20 
detecting the monitored event by the monitored 
event detecting engine according to an embodi- 
ment of the present invention; 
Fig. 13 is a view for illustrating another method of 
setting areas concerned in the course of executing 25 
the procedure for detecting the monitored event, 
which method is carried out by the monitored event 
detecting engine according to an embodiment of 
the present invention: 

Fig. 1 4 is a diagram for illustrating a data structure so 
of a monitored event database according to an 
embodiment of the present invention; 
Fig. IS is a view for illustrating algorithm for deter- 
mining a representative image in an event occur- 
rence period according to an embodiment of the ss 
present invention; 

Figs. 16A, 16B and 16C are views lor illustrating a 
scheme for realizing by HTML a monitoring picture 
display as presented on a user interface by a WEB 
browser according to an embodiment of the present *o 
invention; 

Fig. 17 is a flow chart for illustrating briefly a 
processing flow which is started upon clicking of a 
representative image on a list image display screen 
and which ends when an image is reproduced in an *s 
image reproduction display field according to an 
embodiment of the present invention; 
Fig. 18 is a flow chart for illustrating a procedure for 
generating HTML text data on a list image display 
screen according to an embodiment of the present so 
invention; 

Figs. 19A and 19B are views for illustrating a 
method of dynamically changing over a time tree 
display screen image to a list display screen image 
according to an embodiment of the present inven- ss 
tion; and 

Fig. 20 is a flow chart for illustrating operations acti- 
vated upon clicking of a branch node in a time tree 



structure display according to an em b o d imenl of 
the present invention. 

DESCRIPTION OF THE EMBODIMENTS 

[0026] The present invention wfll be described in 
detail in conjunction with what is presently considered 
as preferred or typical embodiments thereof by refer- 
ence to the drawings. In the following description. Cks 
reference characters designate like or corresponding 
parts throughout the several views. In the following 
description. Dke reference characters designate like or 
corresponding parts throughout the several views. Also 
in the following description, it is to be understood that 
such terms as left", "right", "top", "bottom" and the Eke 
are words of convenience and are not to be construed 
as limiting terms. 

[0027] Figure 1 is a schematic block diagram show- 
ing, by way of example, a system configuration of a 
monitoring system to which an image recorcSngrplay- 
back apparatus according to an exemplary embodiment 
of the present invention can find application. Referring 
to the figure, an image signal issued from a monitoring 
camera 200 is inputted to an image recording^ ayback 
apparatus 100 through an video input terminal 110tobe 
recorded as a digital image signal. The digital image 
signal as recorded can be reproduced and outputted 
through a video output terminal 120 to be displayed on 
a display device 210 in the form of an analogue image. 
Further, the digital image recorded can be sent to a per- 
sonal computer 220 installed at a monitoring center 
from a LAN (Local Area Network) terminal 140 of the 
image recording/playback apparatus 100 via a wide 
area network (WAN) 502. To this end. a telephone line 
or the like can be used. For issuing co m man d s to con- 
trol the image recording/playback apparatus 100, there 
are provided a touch panel 130 and a pointing device 
190. 

[0028] As will be appreciated from the above, a con- 
ventional or existing analogue-type monitoring camera 
and a display device can be used without any substan- 
tial modification as the monitoring devices to be con- 
nected to the image recorcfing^playback apparatus 100 
according to the present invention. At this juncture, it 
should be added that the capability of transmitting the 
monitoring Images to a far-distant place or places via a 
network is a function which can not be found in the con- 
ventional time-lapse video recorder. 
[0029] Figure 2 is a schematic block diagram show- 
ing, by way of example, a system configuration of the 
image recording/playback apparatus according to an 
exemplary embodiment of the present invention. Basi- 
cally, the image recording/playback apparatus is imple- 
mented in a structure which is similar to the system 
corrfigu ration of digital computers employed generally at 
present. 

[0030] Referring to Fig. 2. reference numeral 1 1 1 
denotes an anatogue-to-digital (A/D) converter for con- 
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verting the analogue image signal outputted from the 
monitoring camera into a digital image signal. The dig- 
ital image signal resulting from the A/D conversion is 
stored in a memory 1 70 via an interface 112. in the case 
where ire Image signal outputted from the monitoring 
camera 200 is to be displayed irrtactJy on the display 
device 21 0, the image signal is simultaneously stored in 
a video memory 118. Incidentally, it should be men- 
tioned that the A/D converter 1 1 1 can be spared in the 
case where the digital image signal is inputted from the 
video input terminal 1 10. 

[0031] The video memory 1 18 is arranged to store 
therein the image to be displayed on the display device 
210 as digital data. Reference numeral 119 denotes a 
sort of D/A converter generally known as the RAMDAC 
(RAM Digital-to- Analogue Converter) which is so 
arranged as to read sequentially the data written in the 
video memory 1 18 in conformance with a scanning line 
speed for thereby depicting a relevant image on the dis- 
play device 210. Accordingly every time the data stored 
in the video memory 118 Is updated, the contents 
thereof are immediately reflected En 1he contents of 
image generated on the display device 21 0. Needless to 
say. the display device 210 is a device serving for dis- 
playing the image. In this conjunction, a small-type CRT 
(Cathode Ray Tube), a plasma display, a liquid-crystal- 
type display device or the like may be employed as the 
display device 210. 

[0032] By repeating the image fetching operation 
such as mentioned above at a frequency on the order of 
thirty times per second, there can be generated a dis- 
play of a continuously moving picture on the display 
device. 

[0033] Reference numeral 1 60 denotes an auxiliary 
storage unit which may be constituted by a large capac- 
ity storage such as a hard disk or the like for recording 
thereon the digital data semipermanently. The auxiliary 
storage unit 160 may be so implemented as to be 
removably mounted on the main body of the image 
recording/playback apparatus as a whole such as 
exemplified by a PCMCIA (Personal Computer Memory 
Card International Association)- type hard disk card or 
alternatively the auxiliary storage unit 160 may be con- 
stituted by such type of recording unit in which only the 
recording medium can removably mounted or inserted 
such as exemplified by a DVD-RAM (Digital Video Disk 
Random Access Memory), a flash memory or the like. 
[0034] Reference numeral 150 designates a CPU 
(Central Processing Unit) which is adapted to execute a 
software program or programs for realizing various 
functions such as detection of monitored events (also 
referred to as the events concerned for monitoring or 
concerned events), controls for image recording, selec- 
tion/layout of images to be displayed and others which 
will be described hereinafter in accordance with the 
teachings of the present invention. The software pro- 
gram is resident on the memory 170 in which data 
required for execution of the program may also be 



stored as occasion requires. 

[0035] Reference numeral 139 designates an NIC 
(Network Interface Card) which is designed for transmit- 
ting recorded images and monitored event information 

5 via a network. 

[0036] Commands issued by the user is supplied lo 
the CPU 150 from a touch panel 130 via an interface 
131 to be processed appropriately by the CPU 150. At 
this juncture, it should however be added that in place of 

to the touch panel 1 30, an infrared communication system 
may be adopted for allowing 1he information input oper- 
ation to be performed with the aid of a remote controller. 
[0037] Reference numeral 180 designates a data 
bus for interconnecting the units or devices mentioned 

is above. 

[0038] Now, description will be directed to operation 
of the image recording/playback apparatus imple- 
mented in the system configuration described above. 
[0039] Firstly, description will turn to the operations 

20 involved in recording of the monitoring images (i e , 
images concerned for monitoring). 
[0040] The recording operation of the image record- 
ing/playback apparatus is controlled by a control pro- 
gram 171 which is stored in the memory 1 70 and which 

25 is executed by the CPU 150. At first, the image signal of 
a predetermined time duration is fetched through the 
video input terminal 1 1 0 to be temporarily stored in the 
memory 170 as a train of digital frame images. When 
the image signal as fetched is to be displayed intadty or 

30 straightforwardly on the display device 210. the image 
signal is transferred sequentially to the video memory 
1 18 at the same time as the storage in the memory 1 70, 
to be thereby fed to the display device 210 via the A/D 
converter 1 1 9 and the video output terminal 1 20 for dis- 
ss play as the input image. Subsequently, from the frame 
image train stored temporarily in the memory 1 70, mon- 
itored event information 174 is detected to be stored in 
the memory 1 70. On the basis of the monitored event 
information 174 and control data 172. only the images 

40 which are to be recorded are selectively determined. 
Finally, the selected images and the relevant monitored 
event information are stored in the auxffiary storage unit 
1 60 as the additional image data for the image data 1 61 
and the additional event irrformation for the monitored 

45 event database 162, respectively. In that case, identifi- 
ers (ID) of the events, relevant places and times, char- 
acteristic quantities representing the events, and the 
results of arithmetic operations involved in the selective 
or discriminative decision of the event are stored on an 

so image-by-image basis so that the images can be 
retrieved or searched and/or discrimi natively selected 
on the basis of the types of monitored events and the 
characteristic quantities, 

[0041] Next, description wBI be directed to opera- 
55 tion of the image recording/pl syback apparatus when 
the monitoring images are surveyed or perused by the 
user. Operations involved in the survey or perusal (here- 
inafter referred to simply as the perusal) are executed 
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by the CPU 150 under the control of the control program 
171 which is stored in the memory 170, as in the case 
of the recording operation. At first, in response to ttie 
perusal (survey) request issued by the user and input- 
ted through the medium of the touch panel 130. the s 
monitored event information relevant to the user's 
request is transferred to the memory 170 from the mon- 
itored event database stored in the auxiliary storage unit 
160. In that case, internal records contained in the mon- 
itored event information are read out to thereby delete iq 
those internal records which do not coincide with the 
perusal request from the memory 170. Subsequently, 
the selected internal records contained in the monitored 
event information are read to thereby acquire a path 
name of the image which corresponds to the monitored 75 
event relevant to the user's perusal request, whereon 
the relevant image is transferred to the memory 170 
from the auxiliary storage unit 160 by making use of the 
path name. Subsequently, the monitored event informa- 
tion and the Image In the memory 170 are laid out In an 20 
array easy for observation by the user and sent to the 
video memory 118. The monitored event information 
and the image data transferred to the video memory 
1 18 are displayed intactly on the display device 210 as 
the information indicative of a result of the monitoring Z5 
operation. The operation or processing procedure 
described above can be executed repetitively a number 
of times in response to the input command of the user. 
[0042] In general, in the time-lapse video recorder, 
sampling interval for recording the images can arbitrar- so 
ily be set Needless to say, the image recording over an 
extended time period can be carried out by increasing 
the sampling interval. However, when the sampling 
interval Is simply increased to an excessive extent, 
there may arise such recording omission that an object as 
transversfng the image frame picked up by the camera 
is omitted from recording, giving rise to a problem that 
the intrinsic purpose of the monitoring can not be 
achieved. Thus, according to the teachings of the 
present invention, there is provided a system or scheme *o 
which allows the Image recording to be carried out over 
an extended time period without incurring the recording 
omission, as will be elucidated below. 
[0043] Figure 3. 4 and 5 are views for illustrating a 
system or scheme for recording images relevant to 45 
monitored events as detected. 
[0044] More specifically. Fig. 3 is a view for illustrat- 
ing a recording scheme in which a simple time-lapse 
image recording is combined with filtering effectuated in 
response to monitored events. so 
[0045] Referring to Fig. 3, reference numeral 300 
designates a timing chart for illustrating a timing at 
which a video signal is generated periodically. Thirty 
frames are generated per one second in accordance 
with the NTSC Standards. By contrast, according to the 55 
present invention, thinrting-Out of the frames on a time 
base is realized by fetching the images periodically at a 
predetermined or designated sampling interval. 



[0046] In Fig. 3, reference numeral 310 designates 
a timing chart for the sampling operation, m the case of 
the illustrated example, the frames are thinned out to 
one thirtieth so that one frame is fetched every one sec- 
ond. 

[0047] Reference numeral 320 designates gener- 
ally contents of the Images at the sampling time points, 
respectively. When the monitoring camera is fixed sta- 
tionary with substantially no changes taking place with 
regard to the intruding object and the background 
(frame 320-2), i.e., so long as no monitored event 
occurs. Images of an Input Image train remain essen- 
tially unchanged. Accordingly, when decision is made 
such that no monitored event takes place, i.e., the 
image to be recorded currently is same as a previously 
recorded image, representing the state in which no 
change has occurred with regard to the intruding object 
and the background or when decision is made such that 
the image recorded precedentJy is same as the image to 
be recorded currently, i.e., when no monitored event has 
occurred, then the image recording is omitted. In this 
way, memory consumption brought about by the image 
recording can be suppressed to a minimum. 
[0048] In Fig. 3. frames as selected are illustrated at 
a row 330. As will now be understood, by effectuating 
the filtering operation (I.e.. thinning) In dependence on 
the occurrence of monitored event the memory con- 
sumption to a certain extent win be incurred by the 
recording during a time zone in which the image 
changes at a relatively high frequency. However, when 
ttie image remains less changed as in the case of mon- 
itoring in the nighttime, memory consumption due to 
redundant recording can be suppressed, whereby 
recording over an extended time period can be realized. 
Further, since the images picked up during the time 
period in which the image recording is omitted is same 
as the image of the scene where no change has 
occurred in respect to the intruding object(s) and the 
background or the image recorded immediately before, 
the image search request for the time zone or time point 
at which the Image recording has been omitted can be 
coped with by displaying the correspondent image vice 
the omitted one. 

[0049] The monitoring image information stored in 
the auxiliary storage unit 160 is composed of the image 
data 161 and the monitored event database 162 con- 
taining temporal information indicating the time point at 
which the image data was acquired. Accordingly, in the 
case of the example illustrated in Fig. 3, the search or 
retrieval request to the effect that the image at a time 
point T be displayed can be fulfilled by displaying the 
image at a time point '0* in place of the image at the 
time point "1 ". 

[0050] Figure 4 is a view for illustrating another 
image recording scheme which differs from that illus- 
trated in Fig. 3. 

[0051] Shown at a row 340(b) is a chart for illustrat- 
ing the image sampling timing. In the case of the image 
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recording scheme described previously by reference to 
Fig. 3, decision as to occurrence of the monitored event 
is made after thinning the frames. By contrast in the 
case of the image recording scheme illustrated in Fig. 4, 
the monitored events are detected from all the frames of 5 
the video image inputted directly without performing the 
frame thinning operation, whereon the image recording 
is carried out at a preset sampfing interval. e.g. three 
times per second, after the event period has been deter- 
mined. In that case, by setting the sampling Interval to re 
one frame per thirty seconds which is same as that for 
the input image, it is possible to record the input image 
upon every occurrence of the event In this conjunction, 
it should be mentioned that a plurality of preset image 
recording interval patterns may be changed over in 1S 
dependence on the types or species of the monitored 
events. 

[0052] The image recording scheme illustrated in 
Fig. 4 is suited for the applications where the moving 
direction of a moving object picked up by the monitoring so 
camera or the like Is to be detected as the monitored 
event Besides, this image recording scheme can profit- 
ably be adopted in the image recording in which the 
dynamic image recording interval control described 
below is performed. ss 
[0053] Figures 5A and 5B are views for illustrating 
an image recording scheme in which the dynamic 
image recording interval control is adopted. 
[00S4] Shown in Fig. 5A at a row 350 and in Fig. SB 
at a row 360 are sampfing timing charts, respectively. 30 
[0055] The basic concept underlying the image 
recording now under consideration is essentially same 
as that of the image recording described previously by 
reference to Fig. 4 in that the monitored events are 
detected for recording from all the frames of the input 3s 
video image signal by performing the sample thinning 
operation. The image recording scheme illustrated in 
Figs. 5A and 5B differs from the one Dlustrated in Fig. 4 
in that the sampfing interval is dynamically changed in 
the case of the former by co ntrast to the latter in which 40 
the sampling interval is fixed for the recording. As can 
be seen from the timing chart 350 shown in Fig. 5A, in 
the image recording according to the instant embodi- 
ment of the invention, the monitored event information is 
recorded in linkage with the sampling interval. 45 
[0056] More specifically, changes in a succession of 
image frames are detected as the monitored event To 
this end. absolute difference values of individual pixels 
may be arithmetically determined for the latest frame 
image and the immediately preceding frame image, so 
respectively, whereon a total sum of the absolute differ- 
ence values is determined. The total sum ihus obtained 
is of such nature that ft assumes a large value when the 
image picked up by the camera contains a relatively 
large number of moving objects or when motion of the ss 
moving object(s) is rampant or speedy or when lumi- 
nance of the image frame changes remarkably while 
assuming a value substantially equal to zero when the 



change is insignif icant 

[0057] By taking advantages of the nature men- 
tioned above, it is taught according to the present inven- 
tion to control the sampling interval in dependence on 
magnitude of the above-mention ed value. More specifi- 
cally, when the value is small, the sampling interval is 
increased, e.g. from twice per second to four times or so 
per second, whereas when the value is large, the sam- 
pling interval is decreased to thirty times or so per sec- 
ond. Then, a series of monitoring images as recorded 
can be reproduced, substantially approximating a mov- 
ing picture or motion picture, when many moving 
objects are seen in the images picked up by the moni- 
toring camera or when the object in the picked images 
moves impetuously, whereas when the object(s) picked 
up by the monitoring camera moves only slowly, i.e., 
when no or little change can be seen in the object 
picked up by the monitoring camera, the images output- 
ted therefrom wiD be recorded shatteringfy in an anima- 
tion-like fashion. 

[0058] With this image recording scheme, such 
recording can be realized that the changes of the 
object(s) seen in the image frames remains within a pre- 
determined range Thus, when motion of a person or 
persons are to be recorded, the recording can be per- 
formed for every predetermined motion. 
[0059] Furthermore, there may be established pre- 
viously the rules for controlling of the sampling interval, 
as illustrated in Fig. 5B, being designated by reference 
numeral 360. By way of example, lets suppose a moni- 
toring system for monitoring an entrance of a room 
through which man can enter by opening a door 
installed at the entrance. In that case, it will be important 
that image of the face of a person entering the room can 
be recorded. To this end, the monitored event (change 
in the image) can be detected, for example, when a 
region of flesh-tint exceeds a preset value or range, 
whereon recording is started initially with a short sam- 
pling interval which is then increased progressively. In 
this way, frame images showing the face or complexion 
of the person who is entering the room can be recorded 
without fai. 

[0060] As will now be appreciated, in the image 
recoiding£layback apparatus according to the present 
invention, decision as to execution of recording and con- 
trol of the image recording interval are validated in 
dependence on the monitored events, whereby the 
image recording can be carried out over the extended 
time period without incurring recording omission of the 
event under monitoring such as motion, performance, 
behavior pattern or the like. 

[0061] As a modification of the means for detecting 
the monitored event, such arrangement may be 
adopted that occurrence of the monitored event such as 
"moving object" or "change of 01 umi nation" is deter- 
mined when the number of fields in the input frame 
image which differ significantly from one another 
exceeds a predetermined value 



15 



so 



25 



15 



EP 1 024 668 A2 



[0062] Further, the means lor detecting the moni- 
tored event may be implemented such that when time- 
dependent variations or changes in the number of the 
fields in the input frame image which differ significantly 
from one another exceed a predetermined value, deci- 
sion is made that the monitored event "change of the 
background" has occurred. 

[0063] Furthermore, the means for detecting the 
monitored event may be implemented such that when 
the number of regions or fields in the Input frame image 
which are colored in flesh-tint exceeds a predetermined 
value, decision Is made that the monitored event "face 
candidate field exists" has occurred. 
[0064] In this conjunction, the degree of difference 
among the fields or regions mentioned above may be 
determined, for example, by arithmetically determining 
mean values of the tints or luminances on a block-by- 
block basis from a given image recorded in the past and 
an image inputted currently, respectively, whereon 
absolute values of differences among the correspond- 
ing blocks are determined, respectively. Then, the max- 
imum value of the absolute values can be regarded as 
indicating the degree of difference. 
[0065] Alternatively, the degree of difference may 
be determined as a moving average value of time-series 
values obtained by calculating, respectively, absolute 
values of differences among the corresponding blocks 
in the mean values of tints or luminances on a block-by- 
block basis for the Image frames inputted from a given 
time point in the past up to the current time point 
[0066] Figures 6. 7 and 8 are views for illustrating 
screen images generated upon image recording and 
reproduction in the monitoring image recording/play- 
back apparatus according to an embodiment of the 
present invention. These screen images are displayed 
on the screen of the display device 21 0 which is covered 
with the touch panel 130- 

[0067] More specifically. Fig. 6 shows a screen 
image in the image recording operation mode. Basically, 
the screen image is constituted by a main display field 
400 for displaying the image being inputted currently 
and other fields disposed around the main display fietd 
400. Hereinafter, the main display field 400 will also be 
referred to as the input image field. Displayed on the 
screen image, being superposed, are a monitored event 
information display field 401 indicating r el event location 
and date, etc. and a marker field 402 indicating that a 
monitored event is being currently detected and that the 
recording is being performed currently. In Fig. 6. refer- 
ence numeral 403 designates a button for controlling 
the start of the image monitoring operation of the moni- 
toring system. 

[0068] The monitored event information display 40 1 
may contain identifier (ID) indicating the place or loca- 
tion which is currently photographed by the monitoring 
camera (the name of the monitoring place registered 
previously by the user), date (year: month: day), a time 
code (hour: minute: second and identifier number of the 



frame(s) appearing within a current time period of one 
second) and others. 

[0069] By making use of the marker 402, there can 
be realized a method of messaging the user of the mon- 

s rtored event detection state and the image recording 
state. By way of example, blinking of the marker field 
402 in yellow may be triggered for issuing a message 
that detection of a monitored event is started by the 
monitoring system. Further, momentary blinking of the 

w marker 402 in red in the image recording mode may be 
used as a message indicating that the recording on the 
auxiliary storage unit 160 (i.e.. transfer of the image 
data and the monitored event information data to the 
auxiliary storage unit) is being carried out 

is [0070] Further referring to Fig. 6. reference numeral 
410 denotes a field for messaging the user of history of 
the monitored event as detected. In the case of the illus- 
trated example, a waveform graph 411 "is displayed 
which indicates presence/absence of the monitored 

20 event information and intensity thereof (ag. inter-frame 
difference value). The waveform graph 411 is shifted to 
the left progressively as the time lapses, whereby the 
latest monitored event information value is added in the 
form of waveform value at the right end of the display 

25 field 410. A time indicating display field 412 is disposed 
below the display field 41 0 such that the temporal distri- 
bution of the waveform data 41 1 as a whole can readily 
be comprehended. In the case of the illustrated exam- 
ple, time points preceding to the current time are dis- 
ss played in minute. It should however be understood that 
the absolute time in hour and minute may be displayed. 
[0071 ] Reference numeral 420 designates a display 
field for messaging the user of history of the monitored 
events as detected. In the case of the illustrated exam- 

35 pie, representative images 421 and time points 422 are 
displayed for every monitored event detected up to the 
current time point. More specifically, an image corre- 
sponding to the latest monitored event is displayed at 
the top of the display field 420. which is followed by dis- 

40 plays of representative images 421 and the time points 
422 disposed beneath the latest event image in the tem- 
porally descending order. 

[0072] By clicking graphic portion or image portion 
in the display field 410; 420 with e.g. the pointing device 

4B 1 90, corresponding recorded image can be reproduced. 
[0073] In the playback operation mode, display is 
changed over to the main display field 400 into which 
the recorded image is outputted. as will be described 
later in more detail by reference to Fig. 7. 

so [0074] By way of example, when the representative 
image in the display field 420 is touched or clicked, a 
series of image data 161 generated over a period during 
which the monitored event corresponding to the repre- 
sentative image has occurred are read out from the aux- 

ss ifiary storage unit 160 to be reproduced for display 
within the main display field (input field) 400. On the 
other hand, when the waveform portion in the field 41 0 
is touched, a series of images falling within an event 
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monitoring period closest temporally to the time point 
relevant to the clicked waveform are reproduced in the 
field 400. In synchronism therewith, the representative 
images of the monitored event displayed and the events 
preceding and succeeding, respectively, to the dis- 5 
played event are displayed in the field 420. In that case, 
a vertical Gne of different color (not shown) may be dis- 
played on the touched graph so that the user can readily 
visually identify the touched point 

[0075] In the case of the example described above, to 
the fields 410 and 420 are juxtaposed with the main dis- 
play field 400 In which the picture taten In the recording 
mode and the picture reproduced from the recorded 
images are displayed. It should however be understood 
that the fielde 410 and 420 may be displayed separately is 
from the main display field 400. 
[0076] Figure 7 is a view for illustrating screen 
images generated in the reproduction mode. As can be 
seen in the figure, upon reproduction, a variety of but- 
tons for playback operations are renewedly displayed on 20 
top and bottom portions of the main display field 400. 
Needless to say. these buttons are virtual ones realized 
by combination of graphics displayed on the screen of 
the display device 210 and the touch panel 130. By 
touching these buttons with the pointing device 190 or 2s 
the Oka. operations corresponding to those activated by 
pushing the buttons are carried out by the system. 
[0077] In the main display field 400, there is dis- 
played a moving picture (i.e.. a series of display frames) 
being reproduced in like fashion as the input picture in 30 
the recording operation mode. 
[0078] On the other hand, the monitored event 
information corresponding to the moving picture being 
reproduced is displayed in the monitored event informa- 
tion display 401. ss 
[0079] The monitored event detection marker field 
402 may be displayed equally in the reproduction mode. 
As will be described later on. in the image record- 
ing/playback apparatus according to the present inven- 
tion, recording operation can be performed as the so- <o 
called background operation even in the reproduction 
mode. By way of example, when the user desires to 
reproduce a recorded picture in the course of recording 
the monitoring image, the marker field 402 can be dis- 
played to inform the user of detection of occurrence of 4S 
the monitored event during his or her perusal. Of 
course, the marker field 402 is not displayed so long as 
the image recording is being performed in the back- 
ground. 

[0080] In Fig. 7. reference numeral 430 denotes a so 
button for resuming the recording display state illus- 
trated in Fig. 6. 

[0081] Further, reference numeral 431 denotes a 
display field for messaging the user of a total number of 
frame image data included in the event period of the ss 
monitored event picture being currently reproduced and 
the information concerning the serial frame number of 
the image which is being currently displayed. 



[0082] Reference numeral 432 designates a row of 
buttons assigned, respectively, for forward reproduction 
of picture for display, pause, reverse reproduction, 
frame-by-frame feeding and other operations, respec- 
tively. 

[0083] Finally, reference numeral 433 denotes a 
button for allowing transition or change-over to a repre- 
sentative image list display which wil be described 
below. 

[0084] Figure 8 is a view illustrating a screen image 
for displaying a list of representative images for every 
monitored event as detected. Basically, this screen 
image is substantially same as the display field 420 
shown in Fig. 6 except that the display field is enlarged 
also in the transverse direction, wherein representative 
images 421 and time points 422 are displayed, being 
arrayed in the raster scan sequence. 
[0085] In Fig. 8, triangular buttons 440 disposed at 
top and bottom, respectively; of the display field serve 
for changing over the contents of the list display on a 
time base (along the time axis). More specifically, when 
the top button is pushed, the contents preceding tempo- 
rally by one display unit is displayed on the current 
page. On the other hand, when the bottom button is 
pushed, the contents succeeding temporally by one dis- 
play unit are displayed. When the first or last one of the 
monitored events recorded has been reached through 
user's manipulation, either the top or the bottom button 
which can no more advance the sequence of represent- 
ative images is displayed in faded color, indicating that 
operation of the button in the advancing direction is no 
more possible. 

[0086] Further referring to Fig. 8, when the user 
selects the representative image 421 by touching it with 
the pointing device, the corresponding reproduction 
screen is displayed, as described hereinbefore by refer- 
ence to Rg. 7, whereby the picture of the event monitor- 
ing period as selected is reproduced. 
[0087] Figure 9 is a view for illustrating a user inter- 
face which enables manipulation of the image record- 
ing/playback apparatus 100 and recording, reproduction 
and perusal of the monitoring images at a remote place 
such as a monitoring center. More specifically, this fig- 
ure shows a screen display 500 generated on a display 
of a personal computer 220 installed at a remote moni- 
toring center. 

[0088] In the case of the exemplary embodiment of 
the invention now under consideration, a WEB server 
650 (shown in Fig. 10) and a WEB browser 501 (shown 
in Fig. 10) are employed for realizing control and 
perusal for the image recording/playback apparatus 100 
from the personal computer 220 installed at a remote 
place by way of a network 502. The WEB server is 
instaDed in the image recording/playback apparatus 1 00 
and a WEB page 500 (Rg. 9) on the WEB browser of 
the personal computer 220 located at the remote place 
is displayed, as will be described later on. In the follow- 
ing, description win first be directed to the page men- 
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tioned above. 

[0069] Referring to Fig. 9, a menu 510, a browse 
button 520. a URL (Universal Resource Locator) display 
field 530 and a browser status display field 580 consti- 
tute parts of a WEB browser controlling user interface 5 
for displaying the monitoring information page 500. 
[0090] In addition to the browser components men- 
tioned above, the monitoring information page 500 is 
generally composed of four display fields, l.a. a field 
540 tor displaying the latest or most updated time of the to 
monitoring data, a field 560 for displaying monitored 
events with date/time Information (year, month, 
day/time) for every monitoring place (camera-installed 
location) in a tree structure, a field 570 for displaying in 
a list the representative images for the monitored events rs 
at every designated time (date and hour) and afield 550 
In which a picture for an event monitoring period 
selected in the tree display field 560 or a fist display field 
570 is displayed in the reproduction mode. 
[0091] m the field 540. there is displayed informa- 20 
tion messaging the user of the time at which the moni- 
toring data being displayed currently was pickBd up or 
recorded. In the conjunction, it is noted that with the dia- 
logue facility of such type as the WEB, difficulty is gen- 
erally encountered in sending consecutively and 25 
constantly updated information from the server to the 
browser. In other words, the information can be pre- 
sented to the browser only in a static state preceding to 
a particular time point (because of insufficient band- 
width of the intermediate network such as WAN. high 30 
cost involved for maintaining communication line, heavy 
CPU load imposed on both the server and the clients). 
For these reasons, such situation may arise frequently 
that when a command for perusal of the result of moni- 
toring at a remote place is issued in the course of ss 
recording the monitoring image by the image record- 
ing/playback apparatus 1 00, the page contents become 
frequently different from the state stored internalfy of the 
image recording/playback apparatus 100. Accordingly, it 
is necessary to make the user be conscious of time dif- *o 
ference. The field 540 serves to this end. 
[0092] The field 560 which is designed to display 
the monitored events wrth the date/time information 
"year, month, dayAime" for every m o ni t or ing place (for 
every monitoring camera location) in a tree structure 4S 
provides the interface which allow the user to reach the 
place and/or the date for which the user desires the 
perusal. The tree display is structured hierarchically 
with four levels, i.e.. the place or location 561. date 
(year: month: day) 562, time 563 and the monitored so 
event level 564 as viewed in the descending order. 
When a given one of icons or folders at a given node of 
the tree Is clicked in trie state where the nodes of hierar- 
chically lower level are not displayed, the clicked icon is 
changed ever to the image of opened folder, and the ss 
nodes of lower level are displayed. On the other hand, 
when the nodes of lower level are displayed, the folder 
icon is changed to the image of closed folder with the 



nodes of lower level being hidden. In this way, the user 
can always open only the concerned node for the 
perusal. 

[0093] When a text row 565 at each node is clicked, 
tfie display contents of the image reproduction display 
field 550 and the representative image fist display field 
570 changa By way of example, when the text row 565 
associated with three higher level nodes, e.g. place 561 . 
date (year: month: day) 562 and time 563 are clicked, 
the monitored event list of the latest data/time within the 
range validated by clicking is displayed in the field 570. 
[0094] When the text row of the monitored everrl 
level 564 which is the hierarchically lowest node is 
clicked, then the recorded picture for the relevant event 
monitoring period is reproduced in the image reproduc- 
tion display field 550. 

[0095] in Fig. 9, reference numeral 570 designates 
a field in which a list of representative images for every 
detected monitored events covered by designated time 
periods, respectively, are displayed, similarly to the dis- 
play field 420 shown In Fig. 6. As can be seen In the fig- 
ure, there are displayed In the representative image let 
display field 570 i nformat ion 571 concerning the place 
and the date for the image list being displayed, time 
information 572 indicating the time period on an hour- 
by-hour basis and the representative Images 573. In the 
case where the time information is unavailable, as 
encountered upon activation of the browser, a list of rep- 
resentative images fading within the latest lime period is 
displayed. 

[0096] When a given one of the representative 
images 573 is clicked, a picture (i.e., a series of 
recorded images) recorded during the relevant event 
monitoring period is reproduced in the image reproduc- 
tion display field 550 in an animation -like fashion. 
[0097] in this conjunction, it should be recalled that 
in the case of the conventional time-lapse video 
recorder, the results of monitoring are presented simply 
in the temporal order upon reproduction thereof, which 
thus requires an extremely extensive time. 
[0098] By contrast, by virtue of the user interfaces 
described above by reference to Figs. 6 to 9. the results 
of monitoring can be presented hierarchically in con- 
formance with the temporal structure while the repre- 
sentative images which facflrtate grasping of the 
contents of the monitored events can be presented in a 
two-dimensional array like a tiled image fashion, so to 
say. 

[0099] The tree display makes it possible to make 
access to the concerned date and time at a high speed, 
which has heretofore been impossible with the conven- 
tional time-lapse video recorder. Besides, the picture of 
concern can be reproduced substantially instantly. 
These features are very effective for performing the 
search by figuring out approximately the target Further- 
more, the two-dimensional array of the representative 
images allows the user to survey roughly or browse a 
plurality of monitored events at a glance, whereby the 
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time consumption incurred otherwise, e.g. for searching 
on an image-by-image basis, can be reduced signifi- 
carrtty. 

[01 00] As will now be appreciated, according to the 
teachings of the present invention incarnated in the s 
embodiments described above, the survey or perusal of 
the monitored events as well as confirmation of the con- 
tents thereof can be accomplished with a high speed 
and a high efficiency. In reality, the field test performed 
by the inventors has shown that the time required for the to 
search can be reduced by 1/30 to 1/100. In brief, the 
present invention provides the interface which can 
ensure enhanced usability and advantageous time-sav- 
ing effect 

[0101] Figure 10 is a functional block diagram for is 
illustrating a configuration of the image recording/ play- 
back apparatus 100 from the functional standpoint 
Referring to the figure, the image recording/playback 
apparatus 100 is comprised of five major components. 
They are a main control module 61 0. a monitored event so 
detecting engine 620, a video-output display program 
630. an HTML (TyperText Markup Language) genera- 
tion program 640 and a WEB server 650. 
[0102] The main control module 610 is so designed 
or programmed as to realize the major functions of the zs 
control program 171 described hereinbefore by refer- 
ence to Fig. 2 and in charge of controlling the image 
recording/playback apparatus 100 as a whole. In more 
concrete, the main control module 61 0 detects the user 
commands and the like inputted to the system by so 
means of the touch panel 130 and the WEB server 650 
to thereby control correspondingly operations of the 
monitored event detecting engine 620, the video-output 
display program 630. the HTML generation program 
640 and the WEB server 650. as 
[01 03] The monitored event d steeling engine 620 is 
designed to serve tor detecting the monrtored events 
registered previously from the video signal inputted 
from the monitoring camera 200 to store the rrtonrtoring 
video data and the monitored event information data in 40 
the auxiliary storage unit 160 as the monitoring image 
data 161 and the monitored event database 162, 
respectively. 

[0104] The video-output display program 630 is 
designed to output for display the images, as described 45 
hereinbefore by reference to Fig. 6. More specifically, 
the video-output display program 630 reads out the 
monitoring image data 161 and the monitored event 
database 162 from the auxiliary storage unit 160 in 
response to the user command inputted and transferred so 
to the main control module 610 by way of the touch 
panel 130 to thereby display the relevant image(s) and 
the monitored event information as concerned on the 
screen. 

[0105] The HTML generation program 640 serves ss 
for generating dynamically HTML for outputting the 
WEB page of the monitoring image(s) elucidated previ- 
ously by reference to Fig. 9. In more concrete, manipu- 



lation of the user performed on the remote personal 
computer 220 is transferred to the main control module 
610 from the WEB page 500 through the medium of the 
WEB server 650. In accordance with the user command 
or request inputted to the remote personal computer, 
the monitoring image data 161 and the monitored event 
database 162 are read out from the auxiliary storage 
unit 1 60 to thereby write an HTML tag indicative of a fink 
between the monitored event information and the posi- 
tion of image on the WEB server In the HTML data pre- 
pared in advance. In this way, the dynamic HTML 
generation can be realized. 

[0106] The WEB server 650 serves to transfer the 
WEB page created through the procedure mentioned 
above to the personal computer 220 disposed at a 
remote place and transfer the user request or command 
as Inputted to the main control module 610. 
[0107] Figures 11,12 and 13 are views for illustrat- 
ing a method of detecting the monitored event, which 
method is carried out by the monitored event detecting 
engine 620. 

[0108] tn the following description, it is presumed 
that presence/absence of a moving object is a matter of 
concern for monitoring. 

[0109] At first, individual frame images 700 are 
extracted from the video inputted. Subsequently, the 
frame image is divided into small rectangular areas 710 
for detecting presence/absence of image change in 
each of the small rectangular areas 710, as is illustrated 
in Fig. 11. 

[01 10] As a method or scheme for determining dis- 
criminatively the changes of the image, there may be 
adopted, for example, the method disclosed in U.S. Pat- 
ent No. 5.721 ,692. According to this method, it is possi- 
ble to detect the background, the moving object, 
changes of the background structure and changes of 
illumination in each of the 6ma!l rectangular areas 710 
of concern. 

[0111] tn the following, a moving object detecting 
method in each of these areas will be described briefly. 
[0112] Firstly, the area aimed is selected from the 
frame image and the image of that area at a time point 
where no moving object is observed in that area is set 
as the background image. 

[0113] Subsequently, the degree of d ifference of the 
image is arithmetically determined between the back- 
ground image and the latest one of the image updated 
every input As a typical example of such arithmetic 
determination, differences in brightness of individual 
pixels at corresponding positions in the background 
image and the latest input image are determined, wher- 
eon a sum of squared differences is determined as the 
degree of difference between the background image 
and Ihe inputted image. Alternatively, a sum of squared 
differences in the RGB values may be used as the 
degree of difference. 

[01 1 4] The degrees of difference determined in this 
way are arrayed sequentially along a time axis so that 
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they can be observed in the form of a continuous graph. 
In that case, the graph will exh&rt significant variation 
during the time period in which a moving object is 
detected. By contrast during the time period in which 
no moving object is present, the graph wffl remain flat s 
more or lees with little change. Further, when the back- 
ground of the input image is utterly same as that 
selected as the background image, i.e., so long as the 
contents of the input image are same as the back- 
ground image, the mean value of the degrees of differ- ?0 
ence will remain substantially zero during the 
corresponding time period. On the other hand, when the 
structure or pattern of the background changes due to. 
for example, placement of an article, the mean value will 
then assume a greater value which remains substan- 75 
tally constant during the corresponding time period with 
the graph remaining substantially flat at that value, 
because of change in the contents of the background 
image. 

[0115] By taking advantage of the facts mentioned 20 
above, it is possible to confirm or observe presence/ 
absence of moving objects) and changes of the back- 
ground structure for any given time period. What the 
matter is in this conjunction is how to determine at first 
the image to be adopted as the background image. This zs 
problem can be solved by determining the graph repre- 
senting changes in the degree of time-dependent differ- 
ences of the images picked up appropriately at given 
time points, and by selecting the image at the starting 
time point of the period during which the graph remains so 
flat as the background image. When the structure of the 
background changes in the succeeding processing, 
then the image in the time period during which the 
above-mentioned graph continues to remain substan- 
tially constant after the change of the background image 35 
may be renewed ly set as the background image. The 
background image setting procedure described above 
may be repetitively executed in this manner as occasion 
requires. 

[011 6] Through the procedure described above, the *o 
areas concerned in which the moving object exists can 
be made discriminabfe from those in which no moving 
object exist among the areas set on the frame image by 
coloring or imparting brightness level distinguishable 
from the others, as illustrated in Fig. 1 1 at the right-hand *s 
side. 

[0117] Figure 12 is a view for illustrating a three- 
dimensional time-space image generated by carrying 
out the processing described by reference to Rg. 11 for 
all the frame images and disposing the processed frame so 
images in the time-sequential order. In the three-dimen- 
sional time-space image, the moving object makes 
appearance as a componential cubic image 720, as can 
be seen in Rg. 12. On the basis of the time-space 
image 730, the monitored event i.e., "existence of a £5 
moving object" may be decided in dependence on 
whether the constituent cubic space of a certain size 
can be observed in the time-space image. A graph 740 



illustrated in Rg. 12 beneath the time-space image rep- 
resents occurrence of the monitored event as viewed 
from another standpoint More specifically, the graph 
represents the number of the divided areas in which 
image change has occurred in each image frame Thus, 
the time period 750 shown in Rg. 12 in which the 
graphic curve continues to be substantially constant at a 
value greater than a predetermined one may be 
regarded as the time period in which the monitored 
event has taken place. The time period determined in 
this way is identified or decided discriminativety as the 
event monitoring period during which the moving object 
exists. Thus, the start frame of the event monitoring 
period, the end frame thereof, the length of the time 
period and the results of decisions made on the area- 
by-area bass are stored in the auxfliary storage unit 1 60 
as the monitored event database 162. 
[0118] At this juncture, it should parenthetically be 
mentioned that although the method of deciding the 
event monitoring period by regarding the whole image 
as one area or region is certainly a possble option, the 
procedure of dividing the whole image into small areas 
and identifying the occurrence of event on the basis of 
results of the decision as to change in the property of 
the individual areas is more suited for handling the mon- 
itored events. By way of example, operation of the 
delecting engine can be made robust against external 
disturbances of small magnitudes. Besides, the areas 
excluded from the decision can easily be set Addition- 
ally, the area-based event decision procedure descrfoed 
above by reference to Figs. 1 1 and 12 is advantageous 
in that procedures for the search processing and the 
representative image selection processing (both ol 
which will be described later) can be stated briefly. 
[0119] Detection of other concerned events such as 
changes in the background structure (ag. replacement 
of article(s) or the like in the place being monitored), 
appearance of face areas or the like can be realized 
with enhanced stability on the intra-frame area-by-area 
detection basis, as descnoed previously. 
[0120] In the foregoing description, it has been 
assumed that the areas concerned are each of rectan- 
gular shape. However, the present invention is never 
limited to any particular shape. The area may have any 
other shape. Furthermore, although il has been 
assumed in the foregoing description that the small 
areas are juxtaposed in a regular array pattern (see Rg. 
11), the present invention is never restricted to such 
array. By way of example, a first regular array of small 
areas indicated by phantom line in Fig. 13 may be dis- 
posed in superposition on a second array composed of 
plural areas disposed regularly as indicated by solid line 
in Rg. 13. wherein logic operation may be performed on 
the basis of the result of processings executed on the 
basis of the areas of these two types of arrays. In that 
case, the discriminative decision as to occurrence of the 
event concerned can be realized over an extensive 
monitoring space with high stability and reliability while 



13 



25 



EP 1 024 G66 A2 



26 



reducing effectively the areas for the decision. 
[0121] Figure 14 is a view for illustrating a data 
structure of trie monitored event database 162. 
[01 22] The monitored event database 1 62 is imple- 
mented as an RDB (Relational DataBase) and com- s 
prised of four tables. They are a place table 162-1, an 
image table 162-2, an event table 162-3 and a time tree 
table 162-4. 

[0123] The place table 162-1 contains therein 
records including identifiers (ID) 1 62-1 -1 of the monitor- io 
ing cameras to which the monitoring image record- 
ing/playback apparatus 100 is connected, names 162- 
1-2 of the cameras and names 162-1-3 of the monitor- 
ing places. The place table 162-1 is prepared independ- 
ently in order to reduce redundancy of individual is 
records in the image table 162-2. 
[0124] The Image table 162-2 is provided for the 
purpose of controlling the individual frame images 
stored in the image recording/playback apparatus 100. 
This table contains items of frame image identifier 20 
(record ID numbers), place identifier (ID) 162-2-2. 
recording date (year: minute: second) 162-2-3. record- 
ing time (hour: minute: second) 162-2-4, frame ID 
number 162-2-5. image file path name 162-2-6. event 
species 162-2-7 and event attributes data 162-2-8. 25 
respectively. The place identifier (ID) 162-2-2 is same 
item as the identifier 162-1-3 of the place table 162-1, 
where the actual ID number is stored in both the tables. 
The recording date (year: month: day) 162-2-3. the 
recording time (hour: minute: second) 1 62-2-4 and the 30 
frame ID number 162-2-5 are time information concern- 
ing the data and the time at which the frame image of 
the frame ID number 162-2-5 was recorded. In most of 
the general-purpose relational databases, ttie date data 
(year: month: day) and the time data (hour: minute: sec- as 
ond) are separately handled as the data species capa- 
ble of being handled at high speed. Accordingly, in the 
case of the instant embodiment of the invention, each of 
these data are divided into three data pieces, i.e.. year: 
month: day and hour: minute: second. Stored in the *? 
image table 162-2 at the image file path name 162-2-6 
are path names for the image files stored internally of 
the system to be referenced for reading out actual 
image data upon reproduction and search of the moni- 
tored event image of concern. On the other hand, stored 4s 
in the image table 1 62-2 at the event species 162-2-7 is 
the data indicating what kind of monitored event is 
recorded in the stored frame image. On the presump- 
tion that a plurality of monitored events may possibly 
overlap each other, the species of the monitored events so 
are indicated by bit flags. The event species data 1 62-2- 

7 are referenced upon filtering performed on an event- 
by-event basis for reproduction and retrieval of the mon- 
itoring image. Stored as the event attrbutes data 162-2- 

8 are the characteristic quantity data of the individual ss 
areas on the basis of which the event has been 
detected. 

[0125] The event table 162-3 is referenced upon 



handling of the time period information of the monitored 
events. The event table 162-3 contains event identifier 
(ID) 162-3-1, start frame identifier (ID) 162-3-2, end 
frame identifier (ID) 162-3-3 and representative frame 
identifier (ID) 162-3-4. As the start frame identifier (ID) 
162-3-2 and the end frame identifier (ID) 162-3-3. the 
frame image identifier (ID) 162-2-1 of the image table 
1 62-2 is stored. Similarly, the frame image identifier (ID) 
1 62-2-1 selected by a representative image determining 
algorithm which will be described later on is stored as 
the representative frame identifier (ID) 162-3-4. 
[0126] The time tree table 162-4 is a sort of table 
which is provided for enabling high-speed access to the 
event information with time designation, as in the case 
of the tree display field 560 shown in Rg. 9. The time 
tree table 162-4 is composed of node identifier (ID) 1 62- 
4-1. parent node Identifier (ID) 162-4-2. node search 
key 162-4-3, inter-node sequence 162-4-4. event identi- 
fier (ID) 162-4-5. event species 162-4-6 and place iden- 
tifier (ID) 162-4-7. Although this table is redundant, it is 
provided for the purpose of enhancing the throughput 
upon accessing the results of monitoring. Basically, the 
time tree table 162-4 can dynamically be realized on the 
basis of three tables, i.e.. the place table 162-1, the 
image table 162-2 and the event table 162-3. However, 
arithmetic operation for generating the time tree table 
has to be executed at a very high frequency in the mon- 
itoring picture reproduction/perusal operations. Accord- 
ingly, the time tree table is provided separately in the 
form of table with the aim to reduce the load imposed on 
the system. The node identifier (ID) 162-4-1 and trie 
parent node identifier (ID) 162-4-2 represent fields for 
expressing the tree structure in the form of tables, 
respectively. The node search key 162-4-3 is a charac- 
ter string for accessing directly or straightforwardly the 
node of the lowest level with the aid of information con- 
cerning the place, the date (year: month: day) and the 
time (hour: minute: second) . The parent node identifier 
(ID) 162-4-2 is set as the main key for enabling a hash 
search of the time tree table. Because of the character 
string, key generation is facilitated with the computation 
cost or overhead for the search being reduced. The 
event identifier (ID) 162-4-5 is same as the event identi- 
fier (ID) 1 62-3-1 of the event table 162-3 and represents 
the actual ID number. The event identifier (ID) 162-4-5 
and the place identifier (ID) 162-4-7 are provided for 
carrying out the filtering on the event-by-event basis 
upon reproduction of the monitoring image as well as for 
searching. 

[0127] At this juncture, description wiD briefly be 
made as to the times when the tables data mentioned 
above are generated. The place table 1 62-1 is created 
by the user before the monitoring operation is started. 
The data of the image table 162-2 and the event table 
1 62-3 are dynamically generated in the course of actual 
image recording operation for monitoring. Data of the 
time tree table 162-4 is also generated in the course of 
the monitoring image recording operation. It should 
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however be mentioned that the time tree table 1 62-4 is 
generated secondarily from the contents of the image 
table 162-2 and the event table 162-3. As mentioned 
previously, the time tree table 1 62-4 is prepared in order 
to enhance the throughput upon accessing the results s 
of monitoring operation. 

[01 28] With the data structure described above, the 
image recording/playback apparatus 100 is capable of 
searching the monitored event recorded and the image 
for presentation at high speed. Besides, by virtue of the io 
hardware system structure described hereinbefore by 
reference to Figs. 2 and 10 and the data structure pro- 
vided independently for every individual frame image, a 
novel function of allowing the perusal to be performed 
while performing the monitoring image recording opera- is 
ton simultaneously can be realized. In this conjunction, 
it should be added that such function is Impossible to 
realize with the conventional monitoring/fecording sys- 
tems known heretofore, 

[0129] Figure 15 is a view for illustrating algorithm 20 
or logical procedure for determining the representative 
image in the event monitoring period. 
[0130] It is assumed, by way of example, that the 
event detected by the image recording/playback appa- 
ratus 100 is to be displayed in a same manner as the zs 
main display field 400 (Fig. 6) or the screen display 500 
(Fig. 9). Selection of the image which represents the 
event monitoring period in the case where the event as 
monitored indicates presence of a moving object is an 
important matter which exert significant influence to the so 
efficiency of perusal of the results of the monitoring. 
[01 31 ] By way of example, in case the start frame of 
the event monitoring period is selected as the repre- 
sentative image, such situation will occur frequently that 
only a small part ot the moving object can be observed 35 
and thus the information concerning the detected mov- 
ing object can scarcely be presented. Same holds true 
for the last or end frame. It may be conceived that simply 
an intermediate frame between the start frame and the 
end frame is selected as the representative image. *o 
Such selection of the representative image wfll however 
give rise to a problem in the case where two moving 
objects are detected in succession during one event 
monitoring period. Namely, in this case, the intermedi- 
ate image as selected may be such image as picked up *s 
at a time point when both the moving objects are 
scarcely covered. For the reasons mentioned above, a 
same procedure as that for the control of the sampling 
interval described previously with reference to Figs. 5A 
and 5B is adopted. so 
[0132] In more concrete, it is presumed that upon 
detection of the monitored event, a graphic curve such 
as designated by reference numeral 900 in Fig. 15 is 
obtained. On the presumption, a frame 910-4 at which 
the graphic value assumes a maximum value may be ss 
selected as the representative image, because in that 
case where the event as monitored can be considered 
as indicating the detection of a face of a person. On the 



other hand, in the case the event monitored indicates 
presence of moving objects, there will make appear- 
ance a plurality of local maximum values in the graphic 
curve. In that case, appearance of plural maximum val- 
ues can be deemed to be equivalent to traversal of plu- 
ral moving objects through one event monitoring period. 
In this case, the local maximum value 910-2 appearing 
at first may be selected as the representative image for 
the time being. 

[0133] Next description will turn to a scheme for 
reaEzing by HTML the monitoring picture display as pre- 
sented on the user interface by the WEB browser 
described hereinbefore by reference to Fig. 9. Paren- 
thetically, the technical terms used hereinafter are eluci- 
dated in detail by Uniform Resource Location or URL 
(http://w3c.org/). 

[0134] At first, referring to Figs. 16A to 16C. 
description will be directed to a scheme for realizing 
reproduction of the recorded image of the relevant event 
monitoring period in the image reproduction display field 
550 when the representative image 573 in the repre- 
sentative image list display field 570 is clicked. It is pre- 
sumed that the individual areas are constituted by 
independent HTML display areas which are referred to 
as the frames, respectively. For convenience of the 
description, the representative image list display field 
will be referred to as the "Digest" data with the image 
reproduction display field being termed "Anime" data. 
[0135] Data "Digest" and "Anime" are not static 
data. They are HTML data generated dynamically in 
accordance with the status of the WEB server. For gen- 
erating these dynamic HTML data, server-side scripts 
(i.e., script operating at the WEB server side) such as 
CGI (Common Gateway Interface) and ASP (Active 
Server Pages) are employed. 

[0136] In Fig. 1 6B, reference numeral 1 000 denotes 
an example of HTML text data "Digest". In the case of 
the instant embodiment of the invention. { TABLE > tags 
are used for displaying the character string indicating 
the time point of the monitored event and the represent- 
ative images thereof in tile-pattern arrays, respectively, 
as designated by reference numerals 1010 and 1020, 
respectively. The tag 1030 corresponds to the display 
1010, while the tag 1040 corresponds to the display 
1020. The display image 1020 is an area which can be 
clicked with the pointing device. This can be realized by 
enclosing the image tag 1042 representing the repre- 
sentative Images 573 by using the hypertext tag 
<A> denoted by 1041. By designating the frame 
"Anime" as the destination for link jump and changing 
dynamically the HTML output contents in accordance 
with the clicked image, the images for every corre- 
sponding monitored event can be reproduced. 
[01 37] By changing over the corresponding desti- 
nation in response to the click by the user, the character 
string serving as parameter is written previously in suc- 
cession to a question mark at the jump destination of 
HREF which is an element of the hypertext tag (A) . As 
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designated by reference numeral 1051. the frame 
image identifier (ID) indicating the record range of the 
image table 162-2 or the data capable of extracting the 
frame image identifier (ID) is preferred as the parameter 
to be written In. s 
[0138] Fgure 17 shows a flow chart (denoted gen- 
erally by reference numeral 1 100) for illustrating briefly a 
flow of processings which is started upon clicking of the 
representative image 573 and ends when an image is 
reproduced in the image reproduction display field 550 
in the form of animation. At first when the representa- 
tive Image 573 Is clicked (step 1 1 10). jump to the frame 
"Art me" designated at the WEB browser side is carried 
out (step 1 120). The HTML data displayed as "Anime" 
are dynamically determined and generated from the 
parameter 1051 transferred upon jumping (step 1130). 
In more concrete, in a step 1 132. the frame image iden- 
tifier (ID) indicative of the range on the image table 162- 
2 is extracted from the character string assigned to the 
parameter named "s Jump Key" in a step 11 31. wher- 
eon search of the relevant record Is performed for the 
image table, to thereby extract the individual image path 
names as URLs (Universal Resource Locators) in a 
step 1 133, which is then outputted in the HTML form 
(step 1 134). The WEB browser displays the HTML data 
obtained in this way In the frame "Anime" In a step 1 140. 
Finally, by making the relevant frame image visible or 
invisible in accordance with the user manipulation, the 
frame images are displayed in the form of animation 
(step 1150). 

[0139] Next referring to Fig. 18, description will be 
made of a procedure (generally denoted by reference 
numeral 1200) for generating HTML text data "Digest" 
together with a tree display on the assumption that all 
the data are displayed because of no designation of the 
range tor the list display. In the first place, nodes having 
as the parent node the root node (i.e.. the node for 
which the record ID contains of the time tree are 
searched from an the records in a step 1210. Subse- 
quently, the nodes as searched are sorted with refer- 
ence to the place identifiers (IDs) 162-4-7, respectively, 
in a step 1220. Next the root node is set as the parent 
node (step 1230). After the parent node having been 
determined, the nodes having the same parent node 
identifier (ID) are searched to be sorted in accordance 
with the inter-node order, for thereby generating a list 
(step 1 240). In succession, for the node Est generated in 
this way. move to all the leaves nodes within the tree 
structure is repeated recursively (step 1250). 
[0140] More specifically, it is first checked whether 
or not the current node is a branch node or a leaf node 
(step 1260). When the current node is a branch node, 
then check is made recursively to the node of lower level 
(step 1290). On the other hand, when the current node 
is a leaf node, then the parameter "s Jump Key" is gen- 
erated (step 1270) to be outputted as HTML (step 
1280). 

[0141] A procedure of generating the parameter "s 



Jump Key" will be described below in concrete. In the 
first place, record which coincides with the event identi- 
fier (ID) 162-4-5 oi the node is searched from the event 
table 162-3 (step 1271), to thereby acquire the start 
frame identifier (ID) 1 62-3-2 and the end frame identifier 
(ID) 162-3-3. Subsequently, record which coincides with 
the representative frame identifier (ID) 162-3-4 is 
searched from the image table (step 1273) to thereby 
acquire the representative image path name as the URL 
(step 1274). The processings mentioned above are 
repetitively executed for all the leaves nodes 
[0142] Figures 19A and 1 9B are views for Illustrat- 
ing a method of dynamically changing over the tree dis- 
play field 560 representing a time tree to the list display 
field 570. For convenience of the description, the frame 
name of the tree display field 560 will be referred to as 
"time tree". 

[0143] When the node which is not the leaf in the 
tree display is pointed with the pointing device 190 by 
the user for changing over the display, then the range for 
"data/time" is designated in the frame "Digest". From 
the viewpoint of the internal processing, this corre- 
sponds to the case where the parent node is given con- 
cretely by the time tree table 162-4. Thereafter, HTML 
data are dynamically generated through the same pro- 
cedure as described previously by reference to Fig. 1 8. 
[0144] Referring to Fig. 19B. reference numeral 
1300 denotes generally an example of HTML for realiz- 
ing the jump mentioned above. To this end. a < DIV> tag 
is employed for realizing the tree structure which is 
opened and closed dynarrdcaDy. In order to allow the 
output to be changed over at the destination clicked as 
in the case of the example of "Anime" described previ- 
ously, the value of the parameter "s Digest Key" trans- 
ferred by every hyper-jump tag (A) is changed. As the 
above-mentioned value, the character string constituted 
by "place name" + "date/time" is employed as denoted 
by reference numerals 1310 and 1320. The character 
string is same as the node search key 162-4-3 in the 
time tree table 162-4, which in turn means that the 
structure capable of performing speedily the search for 
depicting the frame "Digest" can be realized. 
[0145] Figure 20 is a flow chart for illustrating oper- 
ations activated upon clicking of a branch node in the 
time tree structure display for the frame "Time Tree". 
[0146] Referring to the figure, when the branch 
node is cEcked (step 1410), jump event to the frame 
"Digest" takes place (step 1420). Because HTML indi- 
cated by the frame "Digest" represents dynamic pages, 
content generation processing is started (step 1430). In 
more concrete, character string value of "6 Digest Key" 
given as the parameter upon jumping is extracted (step 
1431). whereon the coincident record is searched from 
the time tree table 162-4 by using the above-mentioned 
value as the search key (step 1 432). The node identifier 
(ID) 162-4-1 of the record detected in this way is set as 
the parent node, which is then followed by generation of 
the HTML data for the fist display through the same pro- 



13 



20 



25 



30 



35 



AO 



45 



SO 



16 



31 



EP 1 024 666 A2 



cedure as described previously in conjunction with the 
HTM L test data generating procedure 1 200 (step 1 433}. 
Finally, the HTML data is displayed on the frame 
"Digest" in a step 1440. 

[0147] Many modifications and variations of the 
present invention are possible in the light of the above 
techniques. It Is therefore to be understood that within 
the scope of the appended claims, the invention may be 
practiced otherwise than as specifically described. 

Claims 

1 . An image recording/playback apparatus (100) for 

a monitoring system, comprising: 

a first processing unit (150. 170; 610) for 
converting a monitoring image signal (110) 
as inputted into a plurality of time-serial 
digitized monitoring image frames to store 
said digitized image frames in a first stor- 
age area (173); 

a second processing unit (150. 160; 610. 
160) for extracting from said plurality of 
digitized monitoring image frames stored 
in said first storage area a plurality of mon- 
itoring image frames relevant to at least 
one type of concerned event for monitor- 
ing, to thereby store said extracted moni- 
toring image frames in a second storage 
area (161): 

athird processing unit (150, 160; 610, 160) 
for generating a monitored event database 
having a data structure comprised of ele- 
ments each containing information con- 
cerning a storing location of relevant 
monitoring image frame in said second 
storage area and information concerning 
attributes of said relevant monitoring 
image frame for each of the monitoring 
image frames stored in said second stor- 
age area to thereby store said monitored 
event database in a third storage area 
(162); 

a fourth processing unit (150, 130; 610. 
130) responding to a user's perusal 
request designating the element of the 
data structure of said monitored event 
database for acquiring a storing location of 
plural monitoring image frames relevant to 
said perusal request in said second stor- 
age area by searching said monitored 
event database to thereby select from said 
second storage area a plurality of monitor- 
ing image frames relevant to said perusal 
request on the basis of said storing loca- 
tion as acquired; and 

a fifth processing unit (150, 120. 140: 610. 



630. 640. 650) for generating an output 
signal for displaying said selected plural 
monitoring image frames on a display 
device in a two-dimensional array (Fig 8) 
s or in the form of time-serial animation 

(552). 

2. An image recording/playback apparatus for a moni- 
toring system according to claim 1 . 

io wherein said display device (220) is con- 

nected to said image recording/playback apparatus 
via a network (502). 

wherein said fifth processing unit (610. 630. 
640. 650) for generating said output signal includes 

75 a WEB server (650), and 

wherein in response to a perusal request 
Issued by making use of said WEB browser (501) 
by a user viewing display generated on said display 
device, said WEB server (650) sends said plural 

20 monitoring image frames as selected to said dis- 
play device as hypertext markup language (HTML) 
data via said network. (Fig. 1 0) 

3. An image recordingVplayback apparatus for a moni- 
es taring system according to claim 1, wherein said 

fifth processing unit for generating said output sig- 
nal is designed to generate an output 6ignal for dis- 
playing a current monitoring image frame (400) to 
be stored in said first storage area on said display 
so device simuttan eously with said plurafity of monitor- 
ing image frames as selected. 

4. An image receding/playback apparatus for a moni- 
toring system according to claim 1 , 

35 

further comprising: 

a sixth processing unit (150. 160. 170. 120, 
140) tor generating an output signal for dis- 
40 playing elements of the data structure of 

said monitored event database in the form 
of a tree structural expression (560) in 
which said elements are disposed in a tree 
structure. 

4S 

5. An image recording/playback apparatus for a moni- 
toring system according to claim 4, 

wherein the element for which the perusal 
request is issued by the user is designated by click- 
so ing a graphically represented element in said tree 
structural expression being displayed on said dis- 
play device. 

6. An image recording/playback apparatus for a moni- 
55 taring system according to claim 1 , 

wherein said fourth processing unit for 
selecting said monitoring image frames is so desig- 
nated as to search said monitored event database 
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for a predetermined one of the elements of tfie data 
structure of said monitored event database to 
acquire a storing location in said second storage 
area for storing the monitoring image frame rele- 
vant to said predetermined element before said s 
perusal request is issued by the user, for thereby 
selecting the monitoring image frame (421) relevant 
to said predetermined element from said second 
storage area on the bass of said acquired location. 

7. An image recording/playback apparatus for a moni- 
toring system according to claim 6. 

wherein the element for which the perusal 
request is issued by user is designated by clicking 
said selected image frame (421) displayed on said is 
display device means. 

8. An image recording/playback apparatus for a moni- 
toring system according to claim 1 . 

wherein said second and third storage areas so 
are prepared on an auxiliary storage (160). 

9. An image recording/playback apparatus for a moni- 
toring system according to claim 1 , 

wherein the attributes of said monitoring 25 
image frame contain characteristic quantities indic- 
ative of a place where said monitoring image frame 
was taken, a time at which said monitoring image 
frame was taken and a concerned event for moni- 
toring. 30 

10. An image recording/playback method in a monitor- 
ing system, comprising the steps of: 

converting a monitoring image signal (1 10) as as 
inputted into a plurality of time-serial digitized 
monitoring image frames to store said digitized 
image frames in a first storage area (1 73); 
extracting from said plurality of digitized moni- 
toring image frames stored in said first storage 40 
area a plurality of monitoring image frames rel- 
evant to at least one type of concerned event 
for monitoring, to thereby store said extracted 
monitoring image frames in a second storage 
area (161); <s 
generating a monitored event database having 
a data structure comprised of elements each 
containing information concerning a storing 
location of relevant monitoring image frame in 
said second storage area and information con- so 
cerning attributes of said relevant monitoring 
image frame for each of the monitoring image 
frames stored in said second storage area to 
thereby store said monitored event database in 
a third storage area (162); ss 
responding to a user's perusal request desig- 
nating the element of the data structure of said 
monitored event database for acquiring a stor- 



ing location of plural monitoring image frames 
relevant to said perusal request in said second 
storage area by searching said monitored 
event database to thereby select from said sec- 
ond storage area a plurality of monitoring 
image frames relevant to said perusal request 
on the basis of said storing location as 
acquired; and 

generating an output signal for displaying said 
selected plural monitoring image frames on a 
display device in a two-dimensional array (Fig. 
8) or in the form of time-serial animation (552). 

11. An image recording/playback method in a monitor- 
ing system according to claim 10, 

wherein said display device (220) is con- 
nected to said image recording/pl ayback method 
via a network (502), 

wherein in the step of generating said output 
signal, said plural monitoring image frames as 
selected are sent as hypertext markup language 
(HTML) data from a WEB server (650) to said dis- 
play device via said network in response to a 
perusal request issued by making use of a WEB 
browser (501) by a user viewing display generated 
on said display device. (Fig. 10) 

12. An image recording/playback method in a monitor- 
ing system according to claim 10. 

wherein in the step of generating said output 
signal, an output signal is generated for displaying a 
current monitoring image frame (400) to be stored 
in said first storage area on said display device 
simultaneously with said plurality of monitoring 
image frames as selected. 

13. An image recording/playback method in a monitor- 
ing system according to claim 10, 

further comprising the step of: 

generating an output signal for displaying 
elements of the data structure of said mon- 
itored event database in the form of a tree 
structural expression (560) in which said 
elements are disposed in a tree structure. 

14. An image recording/playback method in a monitor- 
ing system according to claim 13, 

wherein the element for which the perusal 
request is issued by the user is designated by click- 
ing a graphically represented element in said tree 
structural expression being displayed on said dis- 
play device. 

15. An image recording/playback method in a monitor- 
ing system according to claim 10, 

wherein in the step of selecting said monitor- 
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ing image frames, said monitored event database is 
searched for a predetermined one of the elements 
of the data structure of said monitored event data- 
base to acquire a storing location in said second 
storage area for storing the monitoring image frame * 
relevant to said predetermined element before said 
perusal request is issued by the user, for thereby 
selecting the monitoring image frame (421) relevant 
to said predetermined element from said second 
storage area on the basis ol said acquired location. 10 

16. An Image recording/playback method in a monitor- 
ing system according to claim 15, 

wherein the element for which the perusal 
request is issued by user is designated by clicking is 
said selected image frame (421) displayed on said 
display device means. 

17. A recording medium storing a computer executable 
program for carrying out an image recording/play- so 
back method for a monitoring system, comprising: 

a first processing module (150. 170; 610) for 
converting a monitoring image signal (110) as 
inputted into a plurality of time-serial digitized 25 
monitoring image frames to store said digitized 
image frames in a first storage area (1 73); 
a second processing module (150, 160; 610. 
160) for extracting from said plurality of digi- 
tized monitoring image frames stored in said 30 
first storage area a plurality of monitoring 
image frames relevant to at least one type of 
concerned event for monitoring, to thereby 
store said extracted monitoring image frames 
in a second storage area (161); as 
a third processing module (150. 160; 610. 160) 
for generating a monitored event database hav- 
ing a data structure comprised of elements 
each containing information concerning a stor- 
ing location of relevant monitoring image frame *o 
in said second storage area and information 
concerning attributes of said relevant monitor- 
ing image frame for each of the monitoring 
image frames stored in said second storage 
area to thereby store said monitored event 45 
database in a third storage area (162); 
a fourth processing module (150, 130; 610. 
130) responding to a user's perusal request 
designating the element of the data structure of 
said monitored event database for acquiring a so 
storing location of plural monitoring image 
frames relevant to said perusal request in said 
second storage area by searching said moni- 
tored event database to thereby select from 
said second storage area a plurality of monitor- 55 
ing image frames relevant to said perusal 
request on the basis of said storing location as 
acquired; and 



a fifth processing module (150, 120. 140; 610, 
630, 640. 650} for generating an output signal 
for displaying said selected plural monitoring 
image frames on a display device in a two- 
dimensional array (Fig. 8) or in the form of time- 
serial animation (552). 

18. An image recording/playback apparatus according 
to claim 1 . 

wherein said second processing unit for 
extracting said monitoring images for storage 
thereof in said second storage area is designed to 
change dynamically an extracting period in depend- 
ence on changes of intensity of the concerned 
event as detected. 

19. An image recording/playback apparatus according 
to claim 6. 

wherein at least one representative monitor- 
ing image of the monitoring image frames relevant 
to each of said predetermined elements is selected 
to be thereby displayed two-dimensionally on said 
display device. 

20. An image recording/playback apparatus according 
to claim 19. 

wherein a first frame image or a last frame 
image or a frame image located at a mid position 
between said first and last frame images within a 
time period during which a concerned event for 
monitoring is occurring is used as said representa- 
tive image. 

21. An image recording/playback apparatus according 
to claim 19, 

wherein a frame image at a position where a 
characteristic quantity representing a concerned 
event is locally maximum or alternatively locally 
minimum or alternatively changing at a high rate is 
employed as said representative image. 

22. An image recording/playback apparatus according 
to claim 1 . 

wherein said plurality of monitoring image 
frames as selected are disposed to be displayed 
rwo-di mans tonally together with time entries on a 
time basis such as date, hour, every fifteen minutes 
and a,m7p.m. 

23. An image recording/playback apparatus according 
to claim 6, 

wherein a characteristic quantity transition 
graph indicating history of the concerned event is 
displayed in association with said selected monitor- 
ing image frame. 

24. An image recording/playback apparatus according 
to claim 2. 
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wherein said fifth processing unit for gener- 
ating said output signal is designed to store the rep- 
resentative images in an image format capable of 
being displayed in a WEB browser on a frame-by- 
frame basis and output an HTML tag ( IMG SRC = s 
image data storage patfi name" > as HTML data for 
displaying said representative image without need 
for copying/moving the image data internally of a 
WEB server (650). 

w 

25. An image recording/playback apparatus according 
to claim 2. 

wherein said fifth processing unit for gener- 
ating said output signal is designed to output an 
HTML tag indicating the representative image or a is 
character string indicating a concerned event as a 
character string sandwiched between (A HREF = 
"another layout display"? parameter) and (/A) for 
thereby allowing said representative image or said 
event to be automatically moved/displayed by click- so 
Ing said sandwiched character string. 

26. An image recording/playback apparatus according 
to claim 25. 

wherein said fifth processing unit for gener- 25 
atlng said output signal Includes means for generat- 
ing dynamically HTML data of event information 
and image on the basis of said event information 
and said image updated every moment. 
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